Multi-lens based capturing apparatus and method

ABSTRACT

A multi-lens based capturing apparatus and method are provided. The capturing apparatus includes a lens array including lenses and a sensor including sensing pixels, wherein at least a portion of sensing pixels in the sensor may generate sensing information based on light entering through different lenses in the lens array, and light incident on each sensing pixel, among the portion of the plurality of sensing pixels may correspond to different combinations of viewpoints.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2016-0158539, filed on Nov. 25, 2016, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference in its entirety.

BACKGROUND 1. Field

Methods and apparatuses consistent with exemplary embodiments relate toa multi-lens based capturing apparatus and method.

2. Description of the Related Art

Due to development of optical technologies and image processingtechnologies, capturing apparatuses are being utilized in a wide rangeof fields, for example, multimedia content, security and recognition.For example, a capturing apparatus may be mounted in a mobile device, acamera, a vehicle or a computer, and may be configured to capture animage, to recognize an object or to acquire data to control a device. Avolume of a capturing apparatus may be determined based on, for example,a size of a lens, a focal length of a lens or a size of a sensor. Forexample, the volume of the capturing apparatus may be adjusted based ona size of a lens or a size of a sensor. As the size of the sensordecreases, an amount of light incident on the sensor may decrease.Accordingly, a resolution of an image may decrease, or it may bedifficult to perform capturing in a low illuminance environment. Toreduce the volume of the capturing apparatus, a multi-lens includingsmall lenses may be used. When the size of the lens decreases, a focallength of the lens may decrease. Accordingly, the volume of thecapturing apparatus may be reduced based on the multi-lens.

SUMMARY

Example embodiments may address at least the above problems and/ordisadvantages and other disadvantages not described above. Also, theexample embodiments are not required to overcome the disadvantagesdescribed above, and an example embodiment may not overcome any of theproblems described above.

According to an aspect of an exemplary embodiment, there is provided acapturing apparatus including: a lens array having a plurality oflenses; and a sensor having a plurality of sensing pixels, where atleast a portion of the plurality of sensing pixels in the sensor maygenerate sensing information based on light entering through differentlenses in the lens array, and light incident on each sensing pixel,among the portion of the plurality of sensing pixels, may correspond todifferent combinations of viewpoints.

In the capturing apparatus, a number of the plurality of sensing pixelsin the sensor and a number of the plurality of lenses in the lens arraymay be relatively prime.

In the capturing apparatus, a ratio between a number of the plurality ofsensing pixels and a number of the plurality of lenses may be a realnumber.

In the capturing apparatus, a matrix indicating a correspondingrelationship between the portion of the plurality of sensing pixels andthe viewpoints may have a full rank.

The capturing apparatus may further include a processor that may beconfigured to generate a captured image based on the sensing informationand a transformation matrix determined based on a correspondingrelationship between the portion of the plurality of sensing pixels andthe viewpoints.

The processor may be further configured to: generate a sensinginformation matrix based on the sensing information, determine pixelvalues that correspond to the viewpoints based on the sensinginformation matrix and the transformation matrix, the pixel values beingincluded in a pixel value matrix, and generate the captured image basedon the pixel values.

In the capturing apparatus, a focal length of the lens array may bedetermined based on a number of lenses in the lens array.

In the capturing apparatus, a focal length of the lens array maydecrease when a number of lenses in the lens array increases.

According to an aspect of an another exemplary embodiment, there isprovided a capturing method including: acquiring sensing informationfrom a plurality of sensing pixels and generating a captured image basedon the sensing information, where at least a portion of the plurality ofsensing pixels may be configured to generate the sensing informationbased on light entering through different lenses in a lens arraycomprising a plurality of lenses, and light incident on each sensingpixel, among the portion of the plurality of sensing pixels, maycorrespond to different combinations of viewpoints.

In the capturing method, a number of the plurality of sensing pixels anda number of the plurality of lenses in the lens array may be relativelyprime.

In the capturing method, a ratio between a number of the plurality ofsensing pixels and a number of the plurality of lenses may be a realnumber.

In the capturing method, a matrix indicating a correspondingrelationship between the portion of the plurality of sensing pixels andthe viewpoints may have a full rank.

In the capturing method, the generating of the captured image mayinclude generating the captured image based on the sensing informationand a transformation matrix determined based on a correspondingrelationship between the portion of the plurality of sensing pixels andthe viewpoints.

In the capturing method, the generating of the captured image mayinclude generating a sensing information matrix based on the sensinginformation; determining pixel values that correspond to the viewpointsbased on the sensing information matrix and the transformation matrix,the pixel values being included in a pixel value matrix; and generatingthe captured image based on the pixel values.

In the capturing method, a focal length of the lens array may bedetermined based on a number of lenses in the lens array.

In the capturing method, a focal length of the lens array may decreasewhen a number of lenses in the lens array increases.

According to an aspect of an another exemplary embodiment, there isprovided a non-transitory computer-readable storage medium storing aprogram for causing a processor to perform the capturing methodincluding: acquiring sensing information from a plurality of sensingpixels and generating a captured image based on the sensing information,where at least a portion of the plurality of sensing pixels may beconfigured to generate the sensing information based on light enteringthrough different lenses in a lens array comprising a plurality oflenses, and light incident on each sensing pixel, among the portion ofthe plurality of sensing pixels, may correspond to differentcombinations of viewpoints.

According to an aspect of an another exemplary embodiment, there isprovided a capturing apparatus including a processor; and a memoryhaving an instruction readable by a computer, where when the instructionis executed by the processor, the processor may be configured to acquiresensing information from a plurality of sensing pixels and to generate acaptured image based on the sensing information. The sensing informationmay be generated by at least a portion of the plurality of sensingpixels based on light entering through different lenses in a lens arraycomprising a plurality of lenses, where light incident on each sensingpixel, among the portion of the plurality of sensing pixels maycorrespond to different combinations of viewpoints.

In the capturing apparatus, a number of the plurality of sensing pixelsin the sensor and a number of the plurality of lenses in the lens arraymay be relatively prime

In the capturing apparatus, a ratio between a number of the plurality ofsensing pixels and a number of the plurality of lenses may be realnumber.

In the capturing apparatus, a matrix indicating a correspondingrelationship between the portion of the plurality of sensing pixels andthe viewpoints may have a full rank.

The processor may be further configured to generate the captured imagebased on the sensing information and a transformation matrix may bedetermined based on a corresponding relationship between the portion ofthe plurality of sensing pixels and the viewpoints.

The processor may be further configured to: generate a sensinginformation matrix based on the sensing information, determine pixelvalues that correspond to the viewpoints based on the sensinginformation matrix and the transformation matrix, the pixel values beingincluded in a pixel value matrix, and generate the captured image basedon the pixel values.

According to an aspect of an another exemplary embodiment, there isprovided a capturing apparatus including: a lens array comprising aplurality of lenses; and a sensor comprising a plurality of sensingpixels, where a ratio between a number of the plurality of sensingpixels and a number of the plurality of lenses is a real number, and afirst light pattern incident on a first sensing pixel, among theplurality of sensing pixels, is different from a second light patternincident on a second sensing pixel, among the plurality of sensingpixels.

The first light pattern may correspond to a first combination ofviewpoints points and the second light pattern may correspond to asecond combination of viewpoints points.

The first combination of viewpoints may be different from the secondcombination of viewpoints.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects of exemplary embodiments will becomeapparent and more readily appreciated from the following detaileddescription of exemplary embodiments, taken in conjunction with theaccompanying drawings of which:

FIG. 1 is a diagram illustrating a capturing apparatus according to anexemplary embodiment;

FIG. 2 is a diagram illustrating an influence of a size of a lensaccording to an exemplary embodiment;

FIG. 3 is a diagram illustrating an example of light incident on sensingpixels by a lens array according to an exemplary embodiment;

FIG. 4 is a diagram illustrating another example of light incident onsensing pixels by a lens array according to an exemplary embodiment;

FIG. 5 is a diagram illustrating a relationship between a number oflenses and a focal length according to an exemplary embodiment;

FIG. 6 is a diagram illustrating a cross section of a lens array and asensor according to an exemplary embodiment;

FIG. 7 is a diagram illustrating a cross section of a pinhole mask and asensor according to an exemplary embodiment;

FIG. 8 is a diagram illustrating a mobile device according to anexemplary embodiment;

FIG. 9 is a diagram illustrating a curved lens array according to anexemplary embodiment;

FIG. 10 is a diagram illustrating a smart vehicle according to anexemplary embodiment;

FIG. 11 is a block diagram illustrating a capturing apparatus accordingto an exemplary embodiment; and

FIG. 12 is a flowchart illustrating a capturing method according to anexemplary embodiment.

DETAILED DESCRIPTION

The following structural or functional descriptions are exemplary tomerely describe the exemplary embodiments, and the scope of theexemplary embodiments is not limited to the descriptions provided in thepresent specification. Various changes and modifications can be madethereto by those of ordinary skill in the art.

Although terms of “first” or “second” are used to explain variouscomponents, the components are not limited to the terms. These termsshould be used only to distinguish one component from another component.For example, a “first” component may be referred to as a “second”component, or similarly, and the “second” component may be referred toas the “first” component within the scope of the right according to theconcept of the present disclosure.

As used herein, the singular forms are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It shouldbe further understood that the terms “comprises” and/or “comprising,”when used in this specification, specify the presence of statedfeatures, integers, steps, operations, elements, components or acombination thereof, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

Unless otherwise defined herein, all terms used herein includingtechnical or scientific terms have the same meanings as those generallyunderstood by one of ordinary skill in the art. Terms defined indictionaries generally used should be construed to have meaningsmatching with contextual meanings in the related art and are not to beconstrued as an ideal or excessively formal meaning unless otherwisedefined herein.

Hereinafter, exemplary embodiments will be described in detail withreference to the accompanying drawings, and like reference numerals inthe drawings refer to like elements throughout.

FIG. 1 is a diagram illustrating a capturing apparatus according to anexemplary embodiment.

Prior to description of the capturing apparatus, brief description willbe provided on factors used to determine a volume of the capturingapparatus and a quality of an image captured by the capturing apparatusaccording to an exemplary embodiment.

According to an exemplary embodiment, the quality of the image capturedby the capturing apparatus may be determined based on a number ofsensing pixels included in a sensor and an amount of light incident on asensing pixel. For example, a resolution of the image may be determinedbased on the number of the sensing pixels included in the sensor, and asensitivity of the image may be determined based on the amount of lightincident on the sensing pixel. The amount of light incident on thesensing pixel may be determined based on a size of the sensing pixel.When the size of the sensing pixel increases, the amount of lightincident on the sensing pixel and a dynamic range of the sensor mayincrease. Accordingly, a resolution of an image captured by the sensormay increase as the number of the sensing pixels included in the sensorincreases. Also, the sensor may operate advantageously for capturing ofa high sensitivity image even at a low illuminance as a size of asensing pixel increases.

A volume of the capturing apparatus may be determined based on a focallength of a lens. For example, the volume of the capturing apparatus maybe determined based on a gap between the lens and a sensor. Because thesensor needs to be located at the focal length of the lens in order tocollect light refracted by the lens, the sensor and the lens included inthe capturing apparatus may need to be spaced apart from each other bythe focal length of the lens. The focal length of the lens may bedetermined based on a viewing angle of the capturing apparatus and asize of the lens (for example, a radius of an aperture of the lens). Forexample, when the viewing angle is fixed, the focal length may increasein proportion to the size of the lens. Also, the size of the lens may bedetermined based on a size of the sensor. For example, to capture animage within a predetermined range of viewing angles, the size of thelens may need to increase as the size of the sensor increases.

As described above, to increase a sensitivity of an image whilemaintaining a viewing angle and a resolution of the image, the volume ofthe capturing apparatus may be increased. For example, to increase asensitivity of an image while maintaining a resolution of the image, asize of each of sensing pixels included in a sensor may need to increasewhile maintaining a number of the sensing pixels. Accordingly, a size ofthe sensor may increase. In this example, to maintain the viewing angle,a size of a lens may increase as the size of the sensor increases, and afocal length of the lens may increase. Thus, the volume of the capturingapparatus may increase.

To reduce the volume of the capturing apparatus, design schemes ofreducing a size of a sensing pixel while maintaining a resolution of thesensor, or of reducing the resolution of the sensor while maintainingthe size of the sensing pixel may be used. In an example, when the sizeof the sensing pixel is reduced while maintaining the resolution of thesensor, a size of the sensor and the focal length of the lens maydecrease, which may lead to a decrease in the volume of the capturingapparatus. However, in this example, a sensitivity of the image may alsodecrease, and a quality of a low illuminance image may be reduced. Inanother example, when the resolution of the sensor is reduced whilemaintaining the size of the sensing pixel, the size of the sensor andthe focal length of the lens may decrease, which may lead to a decreasein the volume of the capturing apparatus. However, in this example, aresolution of an image may be reduced.

The following exemplary embodiments may provide a technology of reducingthe volume of the capturing apparatus while satisfying a desired viewingangle, a desired resolution and a desired sensitivity. Referring to FIG.1, the capturing apparatus includes a lens array 110 and a sensor 120.The lens array 110 includes lenses, and the sensor 120 includes sensingpixels.

For example, when a size of each of the lenses included in the lensarray 110 decreases, that is, when a number of lenses included in thesame area increases, a focal length of each of the lenses and athickness of the capturing apparatus may decrease. In this example, anoriginal high resolution image may be restored by combining lowresolution images captured by the lens array 110. Thus, a thin cameramay be implemented by dividing the lenses in the lens array 110.

Each of the lenses in the lens array 110 may cover a predetermined areaof the sensor 120 corresponding to a size of each of the lenses. Lightpassing through each of the lenses may be incident on sensing pixelsincluded in the area of the sensor 120. Each of the sensing pixels inthe sensor 120 may generate sensing information based on light passingthrough the lenses. For example, sensing pixels 121 and 122 may generatesensing information based on light entering through a lens 111, and asensing pixel 123 may generate sensing information based on lightentering through the lens 111 and a lens 112. The capturing apparatusmay determine pixel values corresponding to viewpoints included in afield of view (FoV) of the capturing apparatus based on sensinginformation output by the sensor 120, and may generate a captured imagebased on the determined pixel values. In this example, a number ofdistinguishable viewpoints may be determined based on a diversity ofsensing information. When the diversity of sensing informationincreases, the number of distinguishable viewpoints and a resolution ofthe captured image may increase.

The diversity of sensing information may be determined based on acombination of viewpoints represented by light incident on each of thesensing pixels. For example, when light corresponding to a firstviewpoint through a third viewpoint is provided to the sensing pixels121 through 123, the diversity of sensing information may increase, incomparison to when light corresponding to the first viewpoint isprovided to the sensing pixels 121 through 123.

When the diversity of sensing information is sufficiently secured andwhen a full rank relationship between the sensing information and thepixel values corresponding to the viewpoints in the FoV of the capturingapparatus is formed, a captured image corresponding to a maximumresolution of the sensor 120 may be derived. The diversity of sensinginformation may be secured based on parameters of the capturingapparatus, for example, a number of the lenses in the lens array 110 ora number of the sensing pixels in the sensor 120.

FIG. 2 is a diagram illustrating an influence of a size of a lensaccording to an exemplary embodiment. Referring to FIG. 2, an FoVprovided by a lens 210 is the same as an FoV provided by lenses 230, anda number of the lenses 230 is three times the lens 210. A focal lengthof the lenses 230 may be reduced to ⅓ of a focal length of the lens 210.

Light may be incident on sensing pixels A₁, A₂, A₃, A₄, A₅, A₆, A₇, A₈and A₉ through the lens 210. Light corresponding to viewpoints X₁, X₂,X₃, X₄, X₅, X₆, X₇, X₈ and X₉ included in an FoV of a capturingapparatus may be uniformly incident on the sensing pixels A₁ through A₉.For example, light corresponding to the viewpoint X₅ may be incident onthe sensing pixel A₅. Accordingly, a pixel value corresponding to theviewpoint X₅ may be restored based on sensing information output by thesensing pixel A₅.

Light may be incident on sensing pixels B₁, B₂, B₃, B₄, B₅, B₆, B₇, B₈and B₉ through the lenses 230. Light corresponding to viewpoints X₁through X₉ included in the FoV may be superimposedly incident on thesensing pixels B₁ through B₉. For example, light corresponding to theviewpoints X₄ through X₆ may be incident on the sensing pixel B₅.Accordingly, a pixel value corresponding to the viewpoints X₄ through X₆may be restored based on sensing information output by the sensing pixelB₅.

When the lens 210 is used, viewpoints may be precisely identified fromeach other, in comparison to when the lenses 230 are used. For example,a resolution of an image captured using the lens 210 may be higher thana resolution of an image captured using the lens 230. When a multi-lensis used, a reduction in a resolution may need to be prevented byproperly adjusting parameters of the capturing apparatus.

FIG. 3 is a diagram illustrating an example of light incident on sensingpixels by a lens array according to an exemplary embodiment. In FIG. 3,light is incident on sensing pixels S₁, S₂, S₃, S₄, S₅, S₆, S₇, S₈ andS₉ through lenses 310, 320 and 330.

Sensing information generated by the sensing pixels S₁ through S₃,sensing information generated by the sensing pixels S₄ through S₆, andsensing information generated by the sensing pixels S₇ through S₉ mayoverlap. Accordingly, when a capturing apparatus has a structure of FIG.3, it may be difficult for the capturing apparatus to obtain all pixelvalues corresponding to capturing viewpoints X₁, X₂, X₃, X₄, X₅, X₆, X₇,X₈ and X₉, and the capturing apparatus may restore a captured imagebased on a resolution lower than a maximum resolution. The sensinginformation generated by the sensing pixels S₁ through S₉ may berepresented as shown in Equation 1 below.

$\begin{matrix}{\begin{bmatrix}I_{1} \\I_{2} \\I_{3} \\I_{4} \\I_{5} \\I_{6} \\I_{7} \\I_{8} \\I_{9}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1\end{bmatrix}\begin{bmatrix}P_{1} \\P_{2} \\P_{3} \\P_{4} \\P_{5} \\P_{6} \\P_{7} \\P_{8} \\P_{9}\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In Equation 1, I₁ through I₉ denote the sensing information generated bythe sensing pixels S₁ through S₉, and P₁ through P₉ denote the pixelvalues corresponding to the capturing viewpoints X₁ through X₉. When aninverse matrix T⁻¹ of a transformation matrix T for transforming a pixelvalue matrix P to a sensing information matrix I is provided, all thepixel values P₁ through P₉ may be obtained, and accordingly the capturedimage with the maximum resolution may be restored from the sensinginformation I₁ through I₉. For the inverse matrix T⁻¹, thetransformation matrix T may need to have a full rank. Thus, theparameters of the capturing apparatus may be adjusted so that thetransformation matrix T may have a full rank.

FIG. 4 is a diagram illustrating another example of light incident onsensing pixels by a lens array according to an exemplary embodiment. InFIG. 4, light is incident on sensing pixels S₁, S₂, S₃, S₄, S₅, S₆, S₇,S₈ and S₉ through lenses 410 and 420.

When light with different patterns is incident on all sensing pixels, atransformation matrix T may have a full rank. When light incident oneach of the sensing pixels corresponds to different combinations ofviewpoints, the transformation matrix T may have the full rank. Forexample, light incident on the sensing pixel S₁ may correspond to acombination of the viewpoints X₁ and X₂, and light incident on thesensing pixel S₂ may correspond to a combination of the viewpoints X₃and X₄. Accordingly, the light incident on the sensing pixels S₁ throughS₉ may correspond to different combinations of the viewpoints X₁ throughX₉. Sensing information generated by the sensing pixels S₁ through S₉may be represented as shown in Equation 2 below.

$\begin{matrix}{\begin{bmatrix}I_{1} \\I_{2} \\I_{3} \\I_{4} \\I_{5} \\I_{6} \\I_{7} \\I_{8} \\I_{9}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 0 \\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1\end{bmatrix}\begin{bmatrix}P_{1} \\P_{2} \\P_{3} \\P_{4} \\P_{5} \\P_{6} \\P_{7} \\P_{8} \\P_{9}\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

In Equation 2, I₁ through I₉ denote the sensing information generated bythe sensing pixels S₁ through S₉, and P₁ through P₉ denote pixel valuescorresponding to the viewpoints X₁ through X₉. Based on Equation 2, thesame number of the pixel values P₁ through P₉ corresponding to anunknown quantity as a number of different relationships may be secured.Accordingly, in Equation 2, a transformation matrix T may be determinedto have a full rank.

A capturing apparatus may obtain an inverse matrix T⁻¹ of atransformation matrix T for transforming a pixel value matrix P to asensing information matrix I. The capturing apparatus may obtain all thepixel values P₁ through P₉ based on Equation 3, and may restore acaptured image with a maximum resolution based on the pixel values P₁through P₉.P=T ⁻¹ ·I  [Equation 3]

The inverse matrix T⁻¹ may be determined based on a correspondingrelationship between the sensing pixels S₁ through S₉ and the viewpointsX₁ through X₉, and may be stored in advance in the capturing apparatus.The capturing apparatus may determine the pixel value matrix P based onthe stored inverse matrix T⁻¹ and the sensing information matrix I.

When nine sensing pixels and two lenses are provided as shown in FIG. 4,light may be provided to “4.5” pixels per lens. For example, when “4001”sensing pixels and “100” lenses are provided, light may be provided to“40.01” pixels per lens. In this example, a first lens may cover a lensoffset of “0” through “40.01,” a second lens may cover a lens offset of“40.01” through “80.02,” and a third lens may cover a lens offset of“80.02” through “120.03.” Also, a last lens that is a hundredth lens maycover a lens offset of “3960.99” through “4001.” When a number ofsensing pixels and a number of lenses are relatively prime, that is,when a ratio between the number of the sensing pixels and the number ofthe lenses is a real number, not an integer, the transformation matrix Tmay have a full rank.

When the number of the sensing pixels and the number of the lenses arerelatively prime, and when the ratio between the number of the sensingpixels and the number of the lenses is a real number, not an integer, atleast a portion of sensing pixels in a sensor may generate sensinginformation based on light entering through different lenses in a lensarray. For example, the sensing pixel S₅ may generate sensinginformation based on light entering through the lens 410 and lightentering through the lens 420. When at least a portion of the sensingpixels in the sensor generate sensing information based on lightentering through different lenses in the lens array, the transformationmatrix T may have the full rank.

According to an exemplary embodiment, parameters of the capturingapparatus may be determined so that light with different patterns may beincident on all sensing pixels or that light incident on each of thesensing pixels may correspond to different combinations of viewpoints.Also, the parameters of the capturing apparatus may be determined sothat at least a portion of the sensing pixels in the sensor may generatesensing information based on light entering through different lenses inthe lens array. The parameters of the capturing apparatus may include,for example, a number of sensing pixels and a number of lenses. Thenumber of the sensing pixels and the number of the lenses may bedetermined to be relatively prime, or a ratio between the number of thesensing pixels and the number of the lenses may be determined as a realnumber, not an integer. Thus, it is possible to derive a captured imagewith the maximum resolution using a multi-lens.

FIG. 5 is a diagram illustrating a relationship between a number oflenses and a focal length according to an exemplary embodiment.Referring to FIG. 5, a number of lenses 530 is relatively prime to anumber of sensing pixels 540, a number of lenses 550 is relatively primeto a number of sensing pixels 560, and a number of lenses 570 isrelatively prime to a number of sensing pixels 580. Accordingly, by thelenses 530, 550 and 570 and the sensing pixels 540, 560 and 580, atransformation matrix T may have a full rank. When the transformationmatrix T has the full rank, a focal length of a lens array may bedetermined based on a characteristic of an application to which acapturing apparatus is applied. For example, when a sensor of a camerais set and a target thickness to a lens and a viewing angle of thecamera are determined, a number of lenses in the lens array may bedetermined.

In FIG. 5, the same FoV may be provided by a lens 510 and the lenses530, 550, and 570. Each of focal lengths of the lenses 510, 530, 550 and570 may be inversely proportional to a number of the lenses 510, 530,550 and 570. For example, because the number of the lenses 530 is twicea number of the lens 510, a focal length of the lenses 530 may be ½ of afocal length of the lens 510. Similarly, a focal length of the lenses550 may be ¼ of the focal length of the lens 510, and a focal length ofthe lenses 570 may be ⅕ of the focal length of the lens 510.Accordingly, a focal length of a lens array may be adjusted based on anumber of lenses included in the lens array. By adjusting the number ofthe lenses in the lens array, the focal length of the lens array may bereduced, which may realize a thin camera.

FIG. 6 is a diagram illustrating a cross section of a lens array 610 anda sensor 620 according to an exemplary embodiment. In FIG. 6, L denotesa number of lenses included in the lens array 610, and P denotes anumber of sensing pixels included in the sensor 620. Accordingly, P/Ldenotes a ratio between the number P of the sensing pixels and thenumber L of the lenses. Each of the lenses may cover the same number ofsensing pixels as a pixel offset corresponding to P/L. As describedabove, when P/L is a real number, not an integer, a resolution of acaptured image may be maximized. Thus, in a process of designing acapturing apparatus, at least one of P or L may be adjusted so that P/Lmay be a real number, not an integer. In FIG. 6, P and L are “37” and“6,” respectively, and accordingly the resolution of the captured imagemay be maximized.

FIG. 7 is a diagram illustrating a cross section of a pinhole mask 710and a sensor 720 according to an exemplary embodiment. In FIG. 7, Pdenotes a number of sensing pixels included in the sensor 720, and Hdenotes a number of pinholes included in the pinhole mask 710. The abovedescription may be applicable to an optical system in addition to alens. In FIG. 7, a pinhole may be applied instead of a lens. P/H denotesa ratio between the number P of the sensing pixels and the number H ofthe pinholes. Each of the pinholes may cover the same number of sensingpixels as a pixel offset corresponding to P/H. Similarly to the abovedescription, when P/H is a real number, not an integer, a resolution ofa captured image may be maximized. Thus, in a process of designing acapturing apparatus, at least one of P or H may be adjusted so that P/Hmay be a real number, not an integer. In FIG. 7, P and H are “37” and“6,” respectively, and accordingly the resolution of the captured imagemay be maximized.

FIG. 8 is a diagram illustrating a mobile device 800 according to anexemplary embodiment. In FIG. 8, the mobile device 800 includes acapturing apparatus 810. According to an exemplary embodiment, reducingof a volume of the mobile device 800 may be important in a design. Themobile device 800 may be a smartphone as shown in FIG. 8, however, thereis no limitation thereto. The mobile device 800 may include, forexample, a wearable device such as a smart watch, a smart band or smartglasses. According to an exemplary embodiment, reducing a volume of awearable device may be more important in a design. As described above,the capturing apparatus 810 may include a multi-lens, and a thickness ofthe capturing apparatus 810 may be adjusted based on a number ofindividual lenses included in the multi-lens. Thus, the mobile device800 may be free from a limitation of a volume by the thickness of thecapturing apparatus 810.

FIG. 9 is a diagram illustrating a curved lens array 910 according to anexemplary embodiment. The curved lens array 910 may include a pluralityof lenses as described above, and accordingly may be designed to becurved by a connection portion between the lenses. Also, the curved lensarray 910 may be designed so that light incident on each sensing pixelmay correspond to different combinations of viewpoints in a curvedstate. The curved lens array 910 may be used for a curved device, forexample, a curved smartphone.

FIG. 10 is a diagram illustrating a smart vehicle according to anexemplary embodiment. Referring to FIG. 10, the smart vehicle mayinclude capturing apparatuses at points 1010, 1020 and 1030. Thicknessesof the capturing apparatuses may be adjusted based on a number of lensesincluded in a multi-lens, and thus the capturing apparatuses may beinstalled in the smart vehicle instead of hindering a design or safetyaspect.

FIG. 11 is a block diagram illustrating a capturing apparatus 1100according to an exemplary embodiment. Referring to FIG. 11, thecapturing apparatus 1100 includes a processor 1110 and a memory 1120.

The processor 1110 may perform at least one of the above-describedmethods. For example, the processor 1110 may process the above-describedcapturing operation. The processor 1110 may acquire sensing informationfrom sensing pixels and may generate a captured image based on thesensing information. The memory 1120 may store computer-readableinstructions. For example, when instructions stored in the memory 1120are executed by the processor 1110, the processor 1110 may process theabove-described capturing operation. Also, the memory 1120 may storedata for capturing, for example, an inverse matrix determined based on acorresponding relationship between sensing pixels and viewpoints.

The processor 1110 may execute instructions or program codes, or maycontrol the capturing apparatus 1100. The capturing apparatus 1100 maybe connected to an external device (for example, a personal computer(PC) or a network) via an input/output device (not shown), and mayexchange data with the external device. The capturing apparatus 1100 maybe implemented as at least a portion of, for example, a mobile devicesuch as a mobile phone, a smartphone, a personal digital assistant(PDA), a netbook, a tablet computer or a laptop computer, a computingdevice such as a desktop computer, an electronic product such as atelevision (TV), a smart TV or a security device for gate control, and asmart vehicle. The above description is also applicable to the capturingapparatus 1100, and accordingly is not repeated here.

FIG. 12 is a flowchart illustrating a capturing method according to anexemplary embodiment. Referring to FIG. 12, in operation 1210, acapturing apparatus acquires sensing information from sensing pixels. Inoperation 1220, the capturing apparatus generates a captured image basedon the ‘sensing information. At least a portion of the sensing pixelsmay generate the sensing information based on light entering throughdifferent lenses in a lens array. Light incident on each of the sensingpixels may correspond to different combinations of viewpoints. The abovedescription is also applicable to the capturing method of FIG. 12, andaccordingly is not repeated here.

The exemplary embodiments described herein may be implemented usinghardware components, software components, or a combination thereof. Aprocessing device may be implemented using one or more general-purposeor special purpose computers, such as, for example, a processor, acontroller and an arithmetic logic unit, a digital signal processor, amicrocomputer, a field programmable array, a programmable logic unit, amicroprocessor or any other device capable of responding to andexecuting instructions in a defined manner. The processing device mayrun an operating system (OS) and one or more software applications thatrun on the OS. The processing device also may access, store, manipulate,process, and create data in response to execution of the software. Forpurpose of simplicity, the description of a processing device is used assingular; however, one skilled in the art will appreciated that aprocessing device may include multiple processing elements and multipletypes of processing elements. For example, a processing device mayinclude multiple processors or a processor and a controller. Inaddition, different processing configurations are possible, such aparallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently orcollectively instruct or configure the processing device to operate asdesired. Software and data may be embodied permanently or temporarily inany type of machine, component, physical or virtual equipment,non-transitory computer storage medium or device, or in a propagatedsignal wave capable of providing instructions or data to or beinginterpreted by the processing device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. The software and datamay be stored by one or more non-transitory computer readable recordingmediums.

The method according to the above-described example embodiments may berecorded in non-transitory computer-readable storage medium includingprogram instructions to implement various operations which may beperformed by a computer. The non-transitory computer-readable medium mayalso include, alone or in combination with the program instructions,data files, data structures, and the like. The program instructionsrecorded on the non-transitory computer-readable medium may be thosespecially designed and constructed for the purposes of the exemplaryembodiments. Examples of non-transitory computer-readable medium includemagnetic media such as hard disks, floppy disks, and magnetic tape;optical media such as CD ROM discs and DVDs; magneto-optical media suchas optical discs; and hardware devices that are specially configured tostore and perform program instructions, such as read-only memory (ROM),random access memory (RAM), flash memory, and the like. Examples ofprogram instructions include both machine code, such as code produced bya compiler, and files containing higher level code that may be executedby the computer using an interpreter. The described hardware devices maybe configured to act as one or more software modules in order to performthe operations of the above-described example embodiments, or viceversa.

While this disclosure includes exemplary embodiments, it will beapparent to one of ordinary skill in the art that various changes inform and details may be made in these exemplary embodiments withoutdeparting from the spirit and scope of the claims and their equivalents.The exemplary embodiments described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each exemplary embodiment are to be consideredas being applicable to similar features or aspects in other exemplaryembodiments. Suitable results may be achieved if the describedtechniques are performed in a different order, and/or if components in adescribed system, architecture, device, or circuit are combined in adifferent manner and/or replaced or supplemented by other components ortheir equivalents. Therefore, the scope of the disclosure is defined notby the detailed description, but by the claims and their equivalents,and all variations within the scope of the claims and their equivalentsare to be construed as being included in the disclosure.

What is claimed is:
 1. A capturing apparatus comprising: a lens arraycomprising a plurality of lenses; and a sensor comprising a plurality ofsensing pixels, wherein at least a portion of the plurality of sensingpixels in the sensor is configured to generate sensing information basedon light entering through different lenses in the lens array, andwherein light incident on each sensing pixel, among the portion of theplurality of sensing pixels, corresponds to different combinations ofviewpoints.
 2. The capturing apparatus of claim 1, wherein a number ofthe plurality of sensing pixels in the sensor and a number of theplurality of lenses in the lens array are relatively prime.
 3. Thecapturing apparatus of claim 1, wherein a ratio between a number of theplurality of sensing pixels and a number of the plurality of lenses is areal number.
 4. The capturing apparatus of claim 1, wherein a matrixindicating a corresponding relationship between the portion of theplurality of sensing pixels and the viewpoints has a full rank.
 5. Thecapturing apparatus of claim 1, further comprising: a processorconfigured to generate a captured image based on the sensing informationand a transformation matrix determined based on a correspondingrelationship between the portion of the plurality of sensing pixels andthe viewpoints.
 6. The capturing apparatus of claim 5, wherein theprocessor is further configured to: generate a sensing informationmatrix based on the sensing information, determine pixel values thatcorrespond to the viewpoints based on the sensing information matrix andthe transformation matrix, the pixel values being included in a pixelvalue matrix, and generate the captured image based on the pixel values.7. The capturing apparatus of claim 1, wherein a focal length of thelens array is determined based on a number of lenses in the lens array.8. The capturing apparatus of claim 1, wherein a focal length of thelens array decreases when a number of lenses in the lens arrayincreases.
 9. The capturing apparatus of claim 1, wherein light incidenton a first sensing pixel, among the portion of the plurality of sensingpixels, corresponds to a first combination comprising a first view pointand a second viewpoint, and light incident on a second sensing pixel,among the portion of the plurality of sensing pixels, corresponds to asecond combination comprising the second view point and a thirdviewpoint, and wherein the first combination is different from thesecond combination.
 10. A capturing method comprising: acquiring sensinginformation from a plurality of sensing pixels; and generating acaptured image based on the sensing information, wherein at least aportion of the plurality of sensing pixels is configured to generate thesensing information based on light entering through different lenses ina lens array comprising a plurality of lenses, and wherein lightincident on each sensing pixel, among the portion of the plurality ofsensing pixels, corresponds to different combinations of viewpoints. 11.The capturing method of claim 10, wherein a number of the plurality ofsensing pixels and a number of the plurality of lenses in the lens arrayare relatively prime.
 12. The capturing method of claim 10, wherein aratio between a number of the plurality of sensing pixels and a numberof the plurality of lenses is a real number.
 13. The capturing method ofclaim 10, wherein a matrix indicating a corresponding relationshipbetween the portion of the plurality of sensing pixels and theviewpoints has a full rank.
 14. The capturing method of claim 10,wherein the generating of the captured image comprises generating thecaptured image based on the sensing information and a transformationmatrix determined based on a corresponding relationship between theportion of the plurality of sensing pixels and the viewpoints.
 15. Thecapturing method of claim 14, wherein the generating of the capturedimage comprises: generating a sensing information matrix based on thesensing information; determining pixel values that correspond to theviewpoints based on the sensing information matrix and thetransformation matrix, the pixel values being included in a pixel valuematrix; and generating the captured image based on the pixel values. 16.The capturing method of claim 10, wherein a focal length of the lensarray is determined based on a number of lenses in the lens array. 17.The capturing method of claim 10, wherein a focal length of the lensarray decreases when a number of lenses in the lens array increases. 18.A non-transitory computer-readable storage medium storing a program forcausing a processor to perform the method of claim
 10. 19. A capturingapparatus comprising: a processor; and a memory comprising aninstruction readable by a computer, wherein when the instruction isexecuted by the processor, the processor is configured to acquiresensing information from a plurality of sensing pixels and to generate acaptured image based on the sensing information, wherein the sensinginformation is generated by at least a portion of the plurality ofsensing pixels based on light entering through different lenses in alens array comprising a plurality of lenses, and wherein light incidenton each sensing pixel, among the portion of the plurality of sensingpixels corresponds to different combinations of viewpoints.
 20. Thecapturing apparatus of claim 19, wherein a number of the plurality ofsensing pixels and a number of the plurality of lenses in the lens arrayare relatively prime.
 21. The capturing apparatus of claim 19, wherein aratio between a number of the plurality of sensing pixels and a numberof the plurality of lenses is a real number.
 22. The capturing apparatusof claim 19, wherein a matrix indicating a corresponding relationshipbetween the portion of the plurality of sensing pixels and theviewpoints has a full rank.
 23. The capturing apparatus of claim 19,wherein the processor is further configured to generate the capturedimage based on the sensing information and a transformation matrixdetermined based on a corresponding relationship between the portion ofthe plurality of sensing pixels and the viewpoints.
 24. The capturingapparatus of claim 23, wherein the processor is further configured to:generate a sensing information matrix based on the sensing information,determine pixel values that correspond to the viewpoints based on thesensing information matrix and the transformation matrix, the pixelvalues being included in a pixel value matrix, and generate the capturedimage based on the pixel values.
 25. A capturing apparatus comprising: alens array comprising a plurality of lenses; and a sensor comprising aplurality of sensing pixels, wherein a ratio between a number of theplurality of sensing pixels and a number of the plurality of lenses is anon-integer number, and wherein a first light pattern incident on afirst sensing pixel, among the plurality of sensing pixels, is differentfrom a second light pattern incident on a second sensing pixel, amongthe plurality of sensing pixels.
 26. The capturing apparatus accordingto claim 25, wherein the first light pattern corresponds to a firstcombination of viewpoints points and the second light patterncorresponds to a second combination of viewpoints points, wherein thefirst combination of viewpoints is different from the second combinationof viewpoints.